Conditional Expectation

Conditional expectation given a non trivial event

Consider a probability space (\Omega, \sF, P) and an event B \in \sF such that P(B) > 0. Then, the conditional probability of any event A \in \sF given an event B is defined as Consider a random variable X: \Omega\to\R defined on a probability space (\Omega, \sF, P), with distribution function F_X: \R \to [0,1], and a non trivial event B \in \sF such that P(B) > 0.

Definition 1. The conditional distribution of X given event B is denoted by F_{X| B}: \R \to [0,1] and F_{X| B}(x) is defined as the probability of event A_X(x) \triangleq X^{-1}(-\infty, x] conditioned on event B for all x\in\R. That is,

Remark 1. The conditional distribution F_{X|B} is a distribution function. This follows from the fact that (i) F_{X|B} \ge 0, (ii) F_{X|B} is right continuous, (iii) \lim_{x\downarrow -\infty}F_{X|B}(x) = 0 and \lim_{x\uparrow \infty}F_{X|B}(x) = 1.

Remark 2. For a discrete random variable X:\Omega\to\sX, the conditional probability mass function of X given a non trivial event B is given by P_{X|B}(x) = \frac{P(X^{-1}\set{x}\cap B)}{P(B)} for all x \in \sX.

Remark 3. For a continuous random variable X:\Omega\to\R, the conditional density of X given a non trivial event B is given by f_{X|B}(x) = \frac{dF_{X|B}(x)}{dx} for all x \in \R.

Example 2 (Conditional distribution). Consider the probability space (\Omega, \sF, P) corresponding to a random experiment where a fair die is rolled once. For this case, the outcome space \Omega = [6], the event space \sF = \cP([6]), and the probability measure P(\omega) = \frac{1}{6} for all \omega\in \Omega.

We define a random variable X: \Omega\to\R such that X(\omega) = \omega for all \omega \in \Omega, and an event B \triangleq \set{\omega \in \Omega: X(\omega) \le 3} = [3] \in \sF. We note that P(B) = 0.5 and the conditional PMF of X given B is

Definition 3. The conditional expectation of X given event B is given as \E[X|B] \triangleq \int_{x\in\R}xdF_{X|B}(x).

Remark 4. For a discrete random variable X:\Omega\to\sX, the conditional expectation of X given a non trivial event B is given by \E[X|B]= \sum_{x\in\sX}xP_{X|B}(x).

Example 4 (Conditional expectation). For the random variable X and event B defined in Example [exmp:FairDieConditionHalf], the conditional expectation \E[X|B] = 2.

Remark 5. Consider two random variables X,Y defined on this probability space, then for y \in \R such that F_Y(y) > 0, we can define events A_X(x) \triangleq X^{-1}(-\infty, x] and A_Y(y) = Y^{-1}(-\infty, y], such that The key observation is that \set{Y \le y} is a non-trivial event. How do we define conditional expectation based on events such as \set{Y = y}? When random variable Y is continuous, this event has zero probability measure.

Conditional expectation given an event space

Consider random variables X:\Omega\to\R and Y:\Omega\to\R defined on the same probability space (\Omega, \sF, P) such that \E\abs{X} < \infty, and a smaller event space \sG \subset \sF. For each non trivial event G \in \sG, we know how to define the conditional distribution F_{X|G} and \E[X|G]. For any trivial event N \in \sG, these are undefined.

Definition 5. The conditional expectation of random variable X given event space \sG is a random variable \E[X\given \sG]: \Omega \to \R defined on the same probability space, such that

Z \triangleq \E[X\given \sG] is \sG measurable,

for all G\in \sG, we have %\int_AXdP = \E[X\Ind{G}] = \E[Z\Ind{G}], %=\int_AYdP,

\E\abs{Z}<\infty.

Lemma 6. The conditional expectation of X given \sG is an a.s. unique random variable.

Proof. Proof. Consider two random variables Z_1 = \E[X|\sG] and Z_2 = \E[X|\sG]. Then from the definition, Z_1, Z_2 are \sG measurable random variables, and Z_1-Z_2 is also \sG measurable. Therefore, G_n \triangleq \set{Z_1 - Z_2 > \frac{1}{n}} \in \sG and \E[(Z_1-Z_2)\Ind{G_n}] = 0 by definition. It follows from continuity of probability, that P(\lim_n G_n) = 0. Similarly, defining F_n \triangleq \set{Z_2 - Z_1 > \frac 1 n}, we can show that P(\lim_nF_n) = 0. ◻

Example 7 (Conditional expectation as averaging). Consider a random variable X:\Omega\to\R defined on a probability space (\Omega,\sF,P) with \E\abs{X} < \infty, and the coarsest event space \sG = \set{\emptyset, \Omega} \subseteq \sF and finest event space \sF. We observe that \E[X|\sG] = \E X a.s. uniquely, since (i) \E X is a constant and hence \sG measurable, (ii) \E[\E X \Ind{\emptyset}] = \E[X \Ind{\emptyset}] = 0 and \E[\E X \Ind{\Omega}] = \E[X \Ind{\Omega}] = \E X, and (iii) \E\abs{\E X} = \abs{\E X} \le \E \abs{X} < \infty from the Jensen’s inequality.

We also observe that \E[X|\sF] = X a.s. uniquely, since (i) X is \sF measurable random variable, (ii) \E[ X \Ind{G}] = \E[X \Ind{G}] for all events G\in \sF, and (iii) \E\abs{X} < \infty.

Lemma 8. The mean of conditional expectation of random variable X given event space \sG is \E X.

Proof. Proof. From the definition of event space \Omega\in\sG, and from the definition of conditional expectation, we get \E[\E[X\mid \sG]] = \E[\E[X\mid \sG]\Ind{\Omega}] = \E[X\Ind{\Omega}] = \E X. ◻

Definition 9. The conditional expectation of X given Y is a random variable \E[X\given Y] \triangleq \E[X\given\sigma(Y)] defined on the same probability space.

Example 10 (Conditioning on simple random variables). For a simple random variable Y: \Omega \to \sY \subseteq \R defined on the probability space (\Omega, \sF, P), we define fundamental events E_y \triangleq Y^{-1}\set{y} \in \sF for all y \in \sY. Then the sequence of events E \triangleq (E_y \in \sF: y \in \sY) partitions the sample space, and we can write the event space generated by random variable Y as \sigma(Y) = (\cup_{y\in I}E_y: I \subseteq \sY).

For a random variable X: \Omega \to \R defined on the same probability space, the random variable Z \triangleq \E[X\given Y] is \sigma(Y) measurable. Therefore, \E[X| Y] = \sum_{y\in\sY}\alpha_y\Ind{E_y} for some \alpha\in\R^\sY. We verify that Z:\Omega\to\R is a \sigma(Y) measurable random variable, since \sigma(Z) \subseteq \sigma(Y). We can also check that \E\abs{Z} < \infty. Further, we have \E[Z\Ind{E_y}] = \E[X\Ind{E_y}] for any y \in \sY, which implies that \alpha_y = \frac{\E[X\Ind{E_y}]}{P_Y(y)} for any y \in \sY. Notice that

Remark 6. There are three main takeaways from this definition. For a random variable Y, the event space generated by Y is \sigma(Y).

1. The conditional expectation \E[X|Y] = \E[X|\sigma(Y)] and is \sigma(Y) measurable. That is, \E[X|Y] is a Borel measurable function of Y. In particular when Y is discrete, this implies that \E[X|Y] is a simple random variable that takes value \E[X|E_y] when \omega \in E_y, and the probability of this event is P_Y(y). When Y is continuous, \E[X|Y] is a continuous random variable with density f_Y.

2. Expectation is averaging. Conditional expectation is averaging over event spaces. We can observe that the coarsest averaging is \E[X|\set{\emptyset, \Omega}] = \E X and the finest averaging is \E[X|\sigma(X)] = X. Further, \E[X|\sigma(Y)] is averaging of X over events generated by Y. If we take any event A \in \sigma(Y) generated by Y, then the conditional expectation of X given Y is fine enough to find the averaging of X when this event occurs. That is, \E[X\Ind{A}] = \E[\E[X|Y]\Ind{A}].

3. If X \in L^1, then the conditional expectation \E[X|Y] \in L^1.

Conditional distribution given an event space

Definition 11. The conditional probability of an event A\in\sF given event space \sG is defined as P(A\mid \sG) \triangleq \E[\Ind{A}\mid\sG].

Remark 7. From the definition of conditional expectation, it follows that P(A\mid \sG):\Omega\to[0,1] is a \sG measurable random variable, such that \E[\Ind{G}P(A|\sG)] = P(A\cap G) for all G \in \sG, and is uniquely defined up to sets of probability zero.

Example 12. For the trivial sigma algebra \sG =\set{\emptyset ,\Omega}, the conditional probability is the constant function P(A\mid \set{\emptyset ,\Omega})=P (A).

Example 13. If A\in \sG, then P(A\mid\sG)=\Ind{A}.

Definition 14. The conditional distribution of random variable X given sub event space \sG is defined as F_{X \mid \sG}(x)\triangleq P(A_X(x)\mid \sG)\text{ for all }x \in \R.

Remark 8. Recall that F_{X\mid\sG}(x): \Omega\to[0,1] a random variable, for each x\in\R. Further, we observe that F_{X\mid\sG} is monotone nondecreasing in x \in \R, right continuous at all x\in\R, and has limits \lim_{x\downarrow-\infty}F_{X\mid\sG}(x) = 0 and \lim_{x\uparrow\infty}F_{X\mid\sG}(x) = 1. It follows that F_{X\mid\sG}: \Omega \to [0,1]^\R is a random distribution.

Theorem 15. Let g:\R\to\R be a Borel measurable function and \sG be a sub-event space. Then, the conditional expectation \E[g(X)\mid\sG] = \int_{x\in\R}g(x)dF_{X\mid\sG}(x).

Proof. Proof. It suffices to show this for simple random variables X:\Omega\to\sX. Since g is Borel measurable, then g(X) is a random variable. We will show that \E[g(X)\mid\sG] = \sum_{x\in\sX}g(x)P_{X\mid\sG}(x) by showing that it satisfies three properties of conditional expectation. For part (i), we observe that from the definition of conditional probability P_{X\mid\sG}(x) is a \sG-measurable random variable for all x\in\sX, and so is the linear combination \sum_{x\in\sX}g(x)P_{X\mid\sG}(x). For part (ii), we let G \in \sG. Then, it follows from the linearity of expectation and the definition of conditional probability, that For part (iii), it follows from the triangle inequality, the linearity of expectation, and the definition of conditional probability that \E\abs{\sum_{x\in\sX}g(x)P_{X\mid\sG}(x)} \le \sum_{x\in\sX}\abs{g(x)} \E P_{X\mid\sG}(x) = \sum_{x\in\sX}\abs{g(x)}P_X(x) = \E\abs{X} < \infty. ◻

Remark 9. The conditional characteristic function is given by \Phi_{X\mid\sG}(u) = \E[e^{juX}\mid\sG] = \int_{x\in\R}e^{jux}dF_{X\mid\sG}(x).

Definition 16. The conditional distribution of random variable X given random variable Y is defined as F_{X\mid Y}(x) \triangleq P(A_X(x)\mid\sigma(Y)] for all x\in \R.

Example 17 (Conditional distribution given simple random variables). Consider a random variable X: \Omega \to \R and a simple random variable Y:\Omega\to\sY defined on the same probability space. Since random variables F_{X\mid Y}(x) = \E[\Ind{A_X(x)}\mid Y] are \sigma(Y) measurable, they can can be written as F_{X\mid Y}(x) = \sum_{y\in\sY}\beta_{x,y}\Ind{E_y} for some \beta_x\in\R^\sY and E_y = Y^{-1}\set{y} for all y \in \sY. Further, we have \E[F_{X\mid Y}(x)\Ind{E_y}] = \E[\Ind{A_X(x)}\Ind{E_y}] for any y \in \sY, which implies that \beta_{x,y} =\frac{P(A_X(x)\cap E_y)}{P_Y(y)} = F_{X\mid E_y}(x) for any y \in \sY. It follows that F_{X\mid Y} is a \sigma(Y) measurable simple random variable.

Example 18 (Conditional expectation). Consider a random experiment of a fair die being thrown and a random variable X:\Omega\to\R taking the value of the outcome of the experiment. That is, for outcome space \Omega = [6] and event space \sF = \cP(\Omega), we have X(\omega) = \omega with P_X(x) = 1/6 for x \in [6]. Define another random variable Y = \SetIn{X \le 3}. Then the conditional expectation of X given Y is a random variable given by Since P\set{Y = 1} = P\set{Y = 0} = 0.5, it follows that that \E[\E[X|Y]] = \E[X] = 3.5.

Example 19 (Conditional distribution). Consider the zero-mean Gaussian random variable N:\Omega\to\R with variance \sigma^2, and another independent random variable Y \in \set{-1,1} with PMF (1-p, p) for some p \in [0,1]. Let X = Y + N, then the conditional distribution of X given simple random variable Y is where F_{X\mid Y^{-1}(\mu)} is \int_{-\infty}^xe^{-\frac{(t-\mu)^2}{\sigma^2}}dt.

Definition 20. When X, Y are both continuous random variables, there exists a joint density f_{X,Y}(x,y) for all (x, y) \in \R^2. For each y \in \sY such that f_Y(y) > 0, we can define a function f_{X\mid Y}: \R^2 \to \R_+ such that

Exercise 21. For continuous random variables X,Y, show that the function f_{X\mid Y^{-1}(y)} is a density of continuous random variable X for each y\in \R.